Systems and methods for processing objects including space efficient distribution stations and automated output processing

ABSTRACT

A space efficient automated processing system for processing objects is disclosed. The processing system includes an input conveyance system for moving objects from an input area in at least an input conveyance vector that includes an input conveyance horizontal direction component and an input conveyance vertical direction component, a perception system for receiving objects from the input conveyance system and for providing perception data regarding an object, a primary transport system for receiving the object from the perception system and for providing transport of the object along at least a primary transport vector including an primary transport horizontal component and a primary transport vertical component that is generally opposite the input conveyance horizontal direction component, and at least two secondary transport systems, each of which receives the object from the primary transport system and moves the object in either of reciprocal directions.

PRIORITY

The present application is a continuation of U.S. patent applicationSer. No. 16/867,127, filed May 5, 2020, now U.S. Pat. No.11,126,807,issued Sep. 21, 2021, which is a continuation of U.S. patentapplication Ser. No. 16/543,105, filed Aug. 16, 2019, now U.S. Pat. No.10,796,116, issued Oct. 6, 2020, which is a continuation of U.S. patentapplication Ser. No. 15/956,442, filed Apr. 18, 2018, now U.S. Pat. No.10,438,034, issued Oct. 8, 2019, which claims priority to U.S.Provisional Patent Application Ser. No. 62/486,783, filed Apr. 18, 2017,the disclosures of which are hereby incorporated by reference in theirentireties.

BACKGROUND

The invention generally relates to automated, robotic and otherprocessing systems, and relates in particular to automated and roboticsystems intended for use in environments requiring, for example, that avariety of objects (e.g., articles, parcels or packages) be processed,e.g., sorted and/or otherwise distributed to several outputdestinations.

Many object distribution systems receive objects in a disorganizedstream that may be provided as individual objects or objects aggregatedin groups such as in bags, arriving on any of several differentconveyances, commonly a conveyor, a truck, a pallet, a Gaylord, or abin. Each object must then be distributed to the correct destinationcontainer, as determined by identification information associated withthe object, which is commonly determined by a label printed on theobject. The destination container may take many forms, such as a bag ora bin.

The processing of such objects has traditionally been done, at least inpart, by human workers that scan the objects, e.g., with a hand-heldbarcode scanner, and then place the objects at assigned locations. Forexample many order fulfillment operations achieve high efficiency byemploying a process called wave picking. In wave picking, orders arepicked from warehouse shelves and placed at locations (e.g., into bins)containing multiple orders that are sorted downstream. At the processingstage individual objects are identified, and multi-object orders areconsolidated, for example into a single bin or shelf location, so thatthey may be packed and then shipped to customers. The processing (e.g.,sorting) of these objects has traditionally been done by hand. A humansorter picks an object from an incoming bin, finds a barcode on theobject, scans the barcode with a handheld barcode scanner, determinesfrom the scanned barcode the appropriate bin or shelf location for thearticle, and then places the article in the so-determined bin or shelflocation where all objects for that order have been defined to belong.Automated systems for order fulfillment have also been proposed. See forexample, U.S. Patent Application Publication No. 2014/0244026, whichdiscloses the use of a robotic arm together with an arcuate structurethat is movable to within reach of the robotic arm.

Other ways of identifying objects by code scanning either require manualprocessing, or require that the code location be controlled orconstrained so that a fixed or robot-held code scanner (e.g., barcodescanner) can reliably detect it. Manually operated barcode scanners aregenerally either fixed or handheld systems. With fixed systems, such asthose used at point-of-sale systems, the operator holds the object andplaces it in front of the scanner so that the barcode faces the scanningdevice's sensors, and the scanner, which scans continuously, decodes anybarcodes that it can detect. If the object is not immediately detected,the person holding the object typically needs to vary the position orrotation of the object in front of the fixed scanner, so as to make thebarcode more visible to the scanner. For handheld systems, the personoperating the scanner looks for the barcode on the object, and thenholds the scanner so that the object's barcode is visible to thescanner, and then presses a button on the handheld scanner to initiate ascan of the barcode.

Further, many current distribution center sorting systems generallyassume an inflexible sequence of operations whereby a disorganizedstream of input objects is first singulated into a single stream ofisolated objects presented one at a time to a scanner that identifiesthe object. A conveyance element or elements (e.g., a conveyor, a tilttray, or manually movable bins) transport the objects to the desireddestination or further processing station, which may be a bin, a chute,a bag or a conveyor etc.

In conventional parcel sortation systems, human workers or automatedsystems typically retrieve objects in an arrival order, and sort eachobject into a collection bin based on a set of given heuristics. Forinstance, all objects of like type might go to a collection bin, or allobjects in a single customer order, or all objects destined for the sameshipping destination, etc. The human workers or automated systems arerequired to receive objects and to move each to their assignedcollection bin. If the number of different types of input (received)objects is large, a large number of collection bins is required.

Such a system has inherent inefficiencies as well as inflexibilitiessince the desired goal is to match incoming objects to assignedcollection bins. Such systems may require a large number of collectionbins (and therefore a large amount of physical space, large capitalcosts, and large operating costs) in part, because sorting all objectsto all destinations at once is not always most efficient.

Current state-of-the-art sortation systems rely on human labor to someextent. Most solutions rely on a worker that is performing sortation, byscanning an object from an induction area (chute, table, etc.) andplacing the object in a staging location, conveyor, or collection bin.When a bin is full, another worker empties the bin into a bag, box, orother container, and sends that container on to the next processingstep. Such a system has limits on throughput (i.e., how fast can humanworkers sort to or empty bins in this fashion) and on number of diverts(i.e., for a given bin size, only so many bins may be arranged to bewithin efficient reach of human workers).

Other partially automated sortation systems involve the use ofrecirculating conveyors and tilt trays, where the tilt trays receiveobjects by human sortation (human induction), and each tilt tray movespast a scanner. Each object is then scanned and moved to a pre-definedlocation assigned to the object. The tray then tilts to drop the objectinto the location. Further, partially automated systems, such as thebomb-bay style recirculating conveyor, involve having trays open doorson the bottom of each tray at the time that the tray is positioned overa predefined chute, and the object is then dropped from the tray intothe chute. Again, the objects are scanned while in the tray, whichassumes that any identifying code is visible to the scanner.

Such partially automated systems are lacking in key areas. As noted,these conveyors have discrete trays that can be loaded with an object;they then pass through scan tunnels that scan the object and associateit with the tray in which it is riding. When the tray passes the correctbin, a trigger mechanism causes the tray to dump the object into thebin. A drawback with such systems however, is that every divert requiresan actuator, which increases the mechanical complexity and the cost perdivert can be very high.

An alternative is to use human labor to increase the number of diverts,or collection bins, available in the system. This decreases systeminstallation costs, but increases the operating costs. Multiple cellsmay then work in parallel, effectively multiplying throughput linearlywhile keeping the number of expensive automated diverts at a minimum.Such diverts do not ID an object and cannot divert it to a particularspot, but rather they work with beam breaks or other sensors to seek toensure that indiscriminate bunches of objects get appropriatelydiverted. The lower cost of such diverts coupled with the low number ofdiverts keep the overall system divert cost low.

Unfortunately, these systems don't address the limitations to totalnumber of system bins. The system is simply diverting an equal share ofthe total objects to each parallel manual cell. Thus each parallelsortation cell must have all the same collection bins designations;otherwise an object might be delivered to a cell that does not have abin to which that object is mapped. There remains a need for a moreefficient and more cost effective object sortation system that sortsobjects of a variety of sizes and weights into appropriate collectionbins or trays of fixed sizes, yet is efficient in handling objects ofsuch varying sizes and weights.

SUMMARY

In accordance with an embodiment, the invention provides a spaceefficient automated processing system for processing objects. Theprocessing system includes an input conveyance system for moving objectsfrom an input area in at least an input conveyance vector that includesan input conveyance horizontal direction component and an inputconveyance vertical direction component, a perception system forreceiving objects from the input conveyance system and for providingperception data regarding an object, a primary transport system forreceiving the object from the perception system and for providingtransport of the object along at least a primary transport vectorincluding an primary transport horizontal component and a primarytransport vertical component that is generally opposite the inputconveyance horizontal direction component, and at least two secondarytransport systems, each of which receives the object from the primarytransport system and moves the object in either of reciprocal directionsthat are each generally parallel with the input conveyance horizontaldirection component and the primary direction horizontal directioncomponent.

In accordance with another embodiment, the invention provides a methodfor providing space efficient automated processing of objects. Themethod includes the steps of conveying objects on an input conveyancesystem from an input area in at least an input conveyance vector thatincludes an input conveyance horizontal direction component and an inputconveyance vertical direction component, receiving objects from theinput conveyance system and for providing perception data regarding anobject responsive to the object falling in a perception system verticaldirection that is generally opposite in direction to the inputconveyance vertical direction component, transporting objects receivedfrom the perception system, and using a primary transport system, alongat least a primary transport vector including a primary transporthorizontal direction component and a primary transport verticalcomponent that is generally opposite the input conveyance horizontaldirection component, and receiving the object from the primary transportsystem, and moving the object in a direction that is generally parallelwith the input conveyance horizontal direction component and the primarydirection horizontal direction component.

In accordance with yet another embodiment, the invention provides anautomated processing system for processing objects. The automatedprocessing system includes an input conveyance system for moving objectsfrom an input area toward a perception system, the perception system forreceiving objects from the input conveyance system and for providingperception data regarding an object, a primary transport system forreceiving the object from the perception system and for providingtransport of the object along at least a primary transport vector, and adiverter system for providing the object to one of a plurality ofprocessing locations, each processing location including a processingbin or box, wherein each of the processing bins or boxes is provided onat least one input bin conveyor system that is biased to urge theprocessing bins or boxes on the input conveyor system to one side of theinput conveyor system.

In accordance with a further embodiment, the invention provides a methodof processing objects. The method includes the steps of moving objectsfrom an input area using an input conveyance system toward a perceptionsystem, receiving the objects from the input conveyance system and forproviding perception data regarding an object using a primary perceptionsystem, receiving the object from the primary perception system and forproviding transport of the object using a primary transport system alongat least a primary transport vector, and diverting the object to one ofa plurality of processing locations, each processing location includinga processing bin or box, wherein each of the processing bins or boxes isprovided on at least one input bin conveyor system that is biased tourge the processing bins or boxes toward one end of the input conveyorsystem

BRIEF DESCRIPTION OF THE DRAWINGS

The following description may be further understood with reference tothe accompanying drawings in which:

FIG. 1 shows an illustrative diagrammatic front view of an objectprocessing system in accordance with an embodiment of the presentinvention;

FIG. 2 shows an illustrative diagrammatic processing side view of thesystem of FIG. 1 ;

FIG. 3 shows another illustrative diagrammatic rear view of the systemof FIG. 1 ;

FIG. 4 shows an illustrative diagrammatic view of a programmable motiondevice processing station in the system of FIG. 1 ;

FIG. 5 shows an illustrative diagrammatic view of the perception systemof FIGS. 2-4 ;

FIG. 6 shows an illustrative diagrammatic view from the perceptionsystem of FIGS. 2-4 , showing a view of objects to be processed;

FIGS. 7A and 7B show illustrative diagrammatic views of a graspselection process in an object processing system of an embodiment of thepresent invention;

FIGS. 8A and 8B show illustrative diagrammatic views of a grasp planningprocess in an object processing system of an embodiment of the presentinvention;

FIGS. 9A and 9B show illustrative diagrammatic views of a graspexecution process in an object processing system of an embodiment of thepresent invention;

FIG. 10 shows an illustrative diagrammatic front view of a dropperception system of FIG. 1 ;

FIG. 11 shows an illustrative diagrammatic rear view of a dropperception system of FIG. 1 ;

FIGS. 12A-12C show illustrative diagrammatic views of an objectdiverting system of FIG. 1 ;

FIG. 13 shows an illustrative diagrammatic view of a processing sectionin an object processing system in accordance with an embodiment of theinvention wherein an object is placed in a carriage;

FIG. 14 shows an illustrative diagrammatic view of the processingsection of FIG. 13 with the carriage having been moved along its track;

FIG. 15 shows an illustrative diagrammatic view of the processingsection of FIG. 13 with the carriage having transferred its load to adestination bin;

FIGS. 16A and 16B show illustrative diagrammatic views of a bin removalmechanism for use in an object processing system in accordance with anembodiment of the invention;

FIG. 17 shows an illustrative diagrammatic view of the processingsection of FIG. 13 with the carriage having returned to its base, and aremoved destination bin being urged from its location;

FIG. 18 shows an illustrative diagrammatic view of the processingsection of FIG. 13 with the removed destination bin being moved along anoutput conveyor;

FIG. 19 shows an illustrative diagrammatic exploded view of a boxassembly for use as a storage bin or destination bin in accordance withvarious embodiments of the present invention;

FIG. 20 shows an illustrative diagrammatic view of the assembled boxtray assembly of FIG. 19 ;

FIG. 21A-21D show illustrative diagrammatic views of a furtherembodiment of a bin displacement system for use in further embodimentsof the invention;

FIG. 22 shows an illustrative diagrammatic view of a flowchart showingselected processing steps in a system in accordance with an embodimentof the present invention; and

FIG. 23 shows an illustrative diagrammatic view of a flowchart showingbin assignment and management steps in a system in accordance with anembodiment of the present invention;

The drawings are shown for illustrative purpose only.

DETAILED DESCRIPTION

In accordance with an embodiment, the invention provides a spaceefficient automated processing system for processing objects. The systemincludes an input conveyance system, a perception system, a primarytransport system, and at least two secondary transport systems. Theinput conveyance system is for moving objects from an input area in atleast an input conveyance vector that includes an input conveyancehorizontal direction component and an input conveyance verticaldirection component. The perception system is for receiving objects fromthe input conveyance system and for providing perception data regardingan object. The primary transport system is for receiving the object fromthe perception system and for providing transport of the object along atleast a primary transport vector including a primary transporthorizontal component and a primary transport vertical component that isgenerally opposite the input conveyance horizontal direction component.The at least two secondary transport systems each of which receive theobject from the primary transport system and move the object in eitherof reciprocal directions that are each generally parallel with the inputconveyance horizontal direction component and the primary directionhorizontal direction component.

The described systems reliably automate the identification andconveyance of such objects, employing in certain embodiments, a set ofconveyors and sensors and a robot arm. In short, applicants havediscovered that when automating sortation of objects, there are a fewmain things to consider: 1) the overall system throughput (objectssorted per hour), 2) the number of diverts (i.e., number of discretelocations to which an object can be routed), 3) the total area of thesortation system (square feet), and 4) the annual costs to run thesystem (man-hours, electrical costs, cost of disposable components).

Processing objects in a distribution center (e.g., for example, sorting)is one application for automatically identifying and moving objects. Ina shipping distribution center for example, objects commonly arrive intrucks, are conveyed to sortation stations where they are processed,e.g., sorted) according to desired destinations, aggregated in bags, andthen loaded in trucks for transport to the desired destinations. Anotherapplication may be in the shipping department of a retail store or orderfulfillment center, which may require that objects be processed fortransport to different shippers, or to different distribution centers ofa particular shipper. In a shipping or distribution center the objectsmay take form of plastic bags, boxes, tubes, envelopes, or any othersuitable container, and in some cases may also include objects not in acontainer. In a shipping or distribution center the desired destinationis commonly obtained by reading identifying information printed on theobject or on an attached label. In this scenario the destinationcorresponding to identifying information is commonly obtained byquerying the customer's information system. In other scenarios thedestination may be written directly on the object, or may be knownthrough other means.

In accordance with various embodiments, therefore, the inventionprovides a method of taking individual objects from a disorganizedstream of objects, providing a generally singulated stream of objects,identifying individual objects, and processing them to desireddestinations. The invention further provides methods for loading objectsinto the system, for conveying objects from one point to the next, fordetermining grasp locations and grasping objects, for excludinginappropriate or unidentifiable objects, for transferring objects fromone conveyor to another, for aggregating objects and transferring tooutput conveyors, for digital communication within the system and withoutside information systems, human operators and maintenance staff, andfor maintaining a safe environment.

Important components of an automated object identification andprocessing system, in accordance with an embodiment of the presentinvention, include an input conveyance system, a perception system, aprimary transport system, and secondary transport systems. FIG. 1 forexample, shows a system 10 that includes an infeed area 12 into whichobjects may be dumped, e.g., by a dumper or transferred from a Gaylord.An infeed conveyor 14 conveys objects from the infeed area 12 to anintermediate conveyor 16 at a processing station 18. The infeed conveyor14 may include cleats for assisting in lifting the objects from theinput area 12 onto the intermediate conveyor 16.

The processing station 18 also includes a grasp perception system 20that views the objects on the intermediate conveyor 16, and identifiesgrasp locations on the objects. The processing station 18 also includesa programmable motion device 22, such as an articulated arm, and aprimary perception system 24 such as a drop perception unit. The graspperception system 20 surveys the objects to identify objects whenpossible, and to determine good grasp points. The object is then graspedby the device 22, and dropped into the drop perception system 24 toensure that the object is accurately identified. The object then fallsthrough the primary perception system 24 onto a primary transport system26, e.g., a conveyor. The primary transport system 26 carries theobjects past one or more diverters 30, 32 that may be engaged to divertan object off of the primary transport system 26 into any of carriages(when the respective carriage is aligned with the diverter) 34, 36, 38or the input area 12. Each of the carriages 34, 36, 38 is reciprocallymovable along a track that runs between rows of destination stations 130of shuttle sections 132 (as discussed below in more detail).

The flow of objects is diagrammatically shown in FIG. 2 , which showsthat objects move from the infeed area 12 to the intermediate conveyor16. The programmable motion device 22 drops the objects into the dropperception unit 24, and the objects then land on the primary transportsystem 26. The objects are then conveyed by the primary transport system26 to diverters that selectively divert objects to carriages (e.g., 36,38). The carriages bring the objects to one of a plurality ofdestination stations 130 (e.g., a processing box or a processing bin)and drops the object into the appropriate destination station. When adestination station is full or otherwise complete, the destinationstation is moved to an output conveyor.

FIG. 3 shows a rear view of the system of FIG. 1 that more clearly showsthe programmable motion device 22 and the drop perception system 24. Theprimary transport system 26 may be a cleated conveyor and the objectsmay be dropped onto the cleated conveyor such that one object isprovided per cleated section. The speeds of the conveyors 14 and 26 mayalso be controlled to assist in providing a singulated stream of objectsto the diverters 30, 32. With reference again to FIG. 1 , thedestination stations 130 (again, e.g., bins or boxes), are provided ondestination input conveyors 160, 162, which may be gravity fed such thatbins or boxes thereon are biased to move toward the processing station18 (as generally shown by corresponding arrows). The destination outputconveyors 150, 152, 154 may also be gravity fed to permit finished binsor boxes to be provided away from the processing station 18 (again, asgenerally shown by corresponding arrows). In further embodiments, theconveyors 150, 152, 154, 160, 162 may be gravity biased in anydirection, or may be actively power controlled. The system may operateusing a computer processing control system 170 that communicates withthe conveyor control systems, the perception units, the programmablemotion device, the diverters, the box or bin removal systems (asdiscussed below), and any and all sensors that may be provided in thesystem.

With reference to FIG. 4 , the processing station 18 of an embodimentincludes a grasp perception system 20 that is mounted above theintermediate conveyor 16, which provides objects to be processed. Thegrasp perception system 20, for example and with reference to FIG. 5 ,may include (on the underside thereof), a camera 40, a depth sensor 42and lights 44. A combination of 2D and 3D (depth) data is acquired. Thedepth sensor 42 may provide depth information that may be used togetherwith the camera image data to determine depth information regarding thevarious objects in view. The lights 44 may be used to remove shadows andto facilitate the identification of edges of objects, and may be all onduring use, or may be illuminated in accordance with a desired sequenceto assist in object identification. The system uses this imagery and avariety of algorithms to generate a set of candidate grasp locations forthe objects in the bin as discussed in more detail below.

The programmable motion device 22 may include a robotic arm equippedwith sensors and computing, that when combined is assumed herein toexhibit the following capabilities: (a) it is able to pick objects upfrom a singulated stream of objects using, for example, an end effector;(b) it is able to move the object to arbitrary places within itsworkspace; and, (c) it is able to generate a map of objects that it isable to pick, represented as a candidate set of grasp points in theworkcell, and as a list of polytopes enclosing the object in space. Theallowable objects are determined by the capabilities of the roboticsystem. Their size, weight and geometry are assumed to be such that therobotic system is able to pick, move and place them. These may be anykind of ordered goods, packages, parcels, or other articles that benefitfrom automated processing.

FIG. 6 shows a representation of an image detected by the graspperception system 20 as it views objects 50, 52, 54 on the intermediateconveyor 16. Superimposed on the objects 50, 52, 54 (for illustrativepurposes) are anticipated grasp locations 60, 62, 64 of the objects.Note that while candidate grasp locations 60, 62, 64 appear to be goodgrasp locations, other grasp locations may not be good grasp locationsif the location is too near an edge of an object, or if the grasplocation is on a very irregular surface of the object or if the objectis partially obscured by another object. Candidate grasp locations maybe indicated using a 3D model of the robot end effector placed in thelocation where the actual end effector would go to use as a grasplocation as shown in FIG. 6 . Grasp locations may be considered good,for example, if they are close to the center of mass of the object toprovide greater stability during grasp and transport, and/or if theyavoid places on an object such as caps, seams etc. where a good vacuumseal might not be available.

If an object cannot be fully perceived by the detection system, theperception system considers the object to be two different objects, andmay propose more than one candidate grasps of such two differentobjects. If the system executes a grasp at either of these bad grasplocations, it will either fail to acquire the object due to a bad grasppoint where a vacuum seal will not occur (e.g., on the right), or willacquire the object at a grasp location that is very far from the centerof mass of the object (e.g., on the left) and thereby induce a greatdeal of instability during any attempted transport. Each of theseresults is undesirable.

If a bad grasp location is experienced, the system may remember thatlocation for the associated object. By identifying good and bad grasplocations, a correlation is established between features in the 2D/3Dimages and the idea of good or bad grasp locations. Using this data andthese correlations as input to machine learning algorithms, the systemmay eventually learn, for each image presented to it, where to bestgrasp an object, and where to avoid grasping an object.

As shown in FIGS. 7A and 7B, the perception system may also identifyportions of an object that are the most flat in the generation of goodgrasp location information. In particular, if an object includes atubular end and a flat end such as object 50, the system would identifythe more flat end as shown at 58 in FIG. 7B. Additionally, the systemmay select the area of an object where a UPC code appears, as such codesare often printed on a relatively flat portion of the object tofacilitate scanning of the barcode.

FIGS. 8A and 8B show that for each object 80, 82, the grasp selectionsystem may determine a direction that is normal to the selected flatportion of the object 80, 82. As shown in FIGS. 9A and 9B, the roboticsystem will then direct the end effector 84 to approach each object 80,82 from the direction that is normal to the surface in order to betterfacilitate the generation of a good grasp on each object. By approachingeach object from a direction that is substantially normal to a surfaceof the object, the robotic system significantly improves the likelihoodof obtaining a good grasp of the object, particularly when a vacuum endeffector is employed.

The invention provides therefore in certain embodiments that graspoptimization may be based on determination of surface normal, i.e.,moving the end effector to be normal to the perceived surface of theobject (as opposed to vertical or “gantry” picks), and that such grasppoints may be chosen using fiducial features as grasp points, such aspicking on a barcode, given that barcodes are almost always applied to aflat spot on the object. The invention also provides operator assist,where an object that the system has repeatedly failed to grasp has acorrect grasp point identified by a human, as well as operator assist,where the operator identifies bad grasp plans, thus removing them andsaving the time of the system attempting to execute them.

In accordance with various embodiments therefore, the invention furtherprovides a sortation system that may learn object grasp locations fromexperience and human guidance. Systems designed to work in the sameenvironments as human workers will face an enormous variety of objects,poses, etc. This enormous variety almost ensures that the robotic systemwill encounter some configuration of object(s) that it cannot handleoptimally; at such times, it is desirable to enable a human operator toassist the system and have the system learn from non-optimal grasps.

The system optimizes grasp points based on a wide range of features,either extracted offline or online, tailored to the gripper'scharacteristics. The properties of the suction cup influence itsadaptability to the underlying surface, hence an optimal grasp is morelikely to be achieved when picking on the estimated surface normal of anobject rather than performing vertical gantry picks common to currentindustrial applications.

In addition to geometric information the system uses appearance-basedfeatures since depth sensors may not always be accurate enough toprovide sufficient information about graspability. For example, thesystem can learn the location of fiducials such as barcodes on theobject, which can be used as indicator for a surface patch that is flatand impermeable, hence suitable for a suction cup. One such example isshipping boxes and bags, which tend to have the shipping label at theobject's center of mass and provide an impermeable surface, as opposedto the raw bag material which might be slightly porous and hence notpresent a good grasp.

By identifying bad or good grasp points on the image, a correlation isestablished between features in the 2D/3D imagery and the idea of goodor bad grasp points; using this data and these correlations as input tomachine learning algorithms, the system can eventually learn, for eachimage presented to it, where to grasp and where to avoid.

This information is added to experience based data the system collectswith every pick attempt, successful or not. Over time the robot learnsto avoid features that result in unsuccessful grasps, either specific toan object type or to a surface/material type. For example, the robot mayprefer to avoid picks on shrink wrap, no matter which object it isapplied to, but may only prefer to place the grasp near fiducials oncertain object types such as shipping bags.

This learning can be accelerated by off-line generation ofhuman-corrected images. For instance, a human could be presented withthousands of images from previous system operation and manually annotategood and bad grasp points on each one. This would generate a largeamount of data that could also be input into the machine learningalgorithms to enhance the speed and efficacy of the system learning.

In addition to experience based or human expert based training data, alarge set of labeled training data can be generated based on a detailedobject model in physics simulation making use of known gripper andobject characteristics. This allows fast and dense generation ofgraspability data over a large set of objects, as this process is notlimited by the speed of the physical robotic system or human input.

The correct processing destination is determined from the symbol (e.g.,barcode) on the object. It is assumed that the objects are marked in oneor more places on their exterior with a visually distinctive mark suchas a barcode or radio-frequency identification (RFID) tag so that theymay be identified with a scanner. The type of marking depends on thetype of scanning system used, but may include 1D or 2D barcodesymbologies. Multiple symbologies or labeling approaches may beemployed. The types of scanners employed are assumed to be compatiblewith the marking approach. The marking, either by barcode, RFID tag, orother means, encodes a symbol string, which is typically a string ofletters and numbers, which identify the object.

Once grasped, the object may be moved by the programmable motion device22 to a primary perception system 24 (such as a drop scanner). Theobject may even be dropped into the perception system 24. In furtherembodiments, if a sufficiently singulated stream of objects is providedon the intermediate conveyor 16, the programmable motion device may beprovided as a diverter (e.g., a push or pull bar) that diverts objectoff of the intermediate conveyor into the drop scanner. Additionally,the movement speed and direction of the intermediate conveyor 16 (aswell as the movement and speed of infeed conveyor 14) may be controlledto further facilitate providing a singulated stream of objects on theintermediate conveyor 16 adjacent the drop scanner.

As further shown in FIGS. 10 and 11 , the primary perception system 24may include a structure 102 having a top opening 104 and a bottomopening 106, and may be covered by an enclosing material 108. Thestructure 102 includes a plurality of sources (e.g., illuminationsources such as LEDs) 110 as well as a plurality of image perceptionunits (e.g., cameras) 112. The sources 60 may be provided in a varietyof arrangements, and each may be directed toward the center of theopening. The perception units 112 are also generally directed toward theopening, although some cameras are directed horizontally, while othersare directed upward, and some are directed downward. The system 24 alsoincludes an entry source (e.g., infrared source) 114 as well as an entrydetector (e.g., infrared detector) 116 for detecting when an object hasentered the perception system 24. The LEDs and cameras thereforeencircle the inside of the structure 102, and the cameras are positionedto view the interior via windows that may include a glass or plasticcovering (e.g., 118).

An aspect of certain embodiments of the present invention, is theability to identify via barcode or other visual markings of objects byemploying a perception system into which objects may be dropped.Automated scanning systems would be unable to see barcodes on objectsthat are presented in a way that their barcodes are not exposed orvisible. The system 24 therefore is designed to view an object from alarge number of different views very quickly, reducing or eliminatingthe possibility of the system 24 not being able to view identifyingindicia on an object.

Key features in the perception system are the specific design of theperception system so as to maximize the probability of a successfulscan, while simultaneously minimizing the average scan time. Theprobability of a successful scan and the average scan time make up keyperformance characteristics. These key performance characteristics aredetermined by the configuration and properties of the perception system,as well as the object set and how they are marked.

The two key performance characteristics may be optimized for a givenitem set and method of labeling. Parameters of the optimization for asystem include how many scanners, where and in what orientation to placethem, and what sensor resolutions and fields of view for the scanners touse. Optimization can be done through trial and error, or by simulationwith models of the object.

Optimization through simulation employs a scanner performance model. Ascanner performance model is the range of positions, orientations andbarcode element size that an identifying symbol can be detected anddecoded by the scanner, where the barcode element size is the size ofthe smallest feature on the symbol. These are typically rated at aminimum and maximum range, a maximum skew angle, a maximum pitch angle,and a minimum and maximum tilt angle.

Typical performance for camera-based scanners are that they are able todetect symbols within some range of distances as long as both pitch andskew of the plane of the symbol are within the range of plus or minus 45degrees, while the tilt of the symbol can be arbitrary (between 0 and360 degrees). The scanner performance model predicts whether a givensymbol in a given position and orientation will be detected.

The scanner performance model is coupled with a model of where symbolswould expect to be positioned and oriented. A symbol pose model is therange of all positions and orientations, in other words poses, in whicha symbol will expect to be found. For the scanner, the symbol pose modelis itself a combination of an article gripping model, which predicts howobjects will be held by the robotic system, as well as a symbol-itemappearance model, which describes the possible placements of the symbolon the object. For the scanner, the symbol pose model is itself acombination of the symbol-item appearance model, as well as aninbound-object pose model, which models the distribution of poses overwhich inbound articles are presented to the scanner. These models may beconstructed empirically, modeled using an analytical model, orapproximate models may be employed using simple sphere models forobjects and uniform distributions over the sphere as a symbol-itemappearance model.

Following detection by the perception unit 24, the object is nowpositively identified and drops onto the primary transport system 26(e.g., a conveyor). With reference again to FIGS. 1 and 3 , the primarytransport system 26 moves the identified object toward diverters 30, 32that are selectively engageable to divert the object off of the conveyorinto any of carriages 34, 36, 38 or (if the object was not able to beidentified), the object may be either returned to the input area 12 orit may be dropped off of the end of the conveyor 26 into a manualprocessing bin. Each carriage 34, 36, 38 is reciprocally movable amongdestination bins 130 of one of a plurality of destination sections 132.Efficiencies in space may be provided in accordance with certainembodiments by having objects first move from the input area 12 alongthe infeed conveyor 14 in a direction that includes a horizontalcomponent and a vertical component. The object then drops through thedrop scanner 24 (vertically) and lands on the primary transport conveyor26, which also moves the object in a direction that has a horizontalcomponent (opposite in direction to that of the infeed conveyor 14) anda vertical component. The object is then moved horizontally by acarriage 36, 38, and dropped (vertically) above a target destinationstation 130, such as a destination bin.

With reference to FIGS. 12A-12B, a diverter unit (e.g., 32) may beactuated to urge an object (e.g., 35) off of the conveyor 26 into aselected carriage (e.g., 38) that runs along a rail 39 betweendestination locations. The diverter unit may include a pair of paddles31 that are suspended by a frame 33 that permits the paddles to beactuated linearly to move an object off of the conveyor in eitherdirection transverse to the conveyor. Again, with reference to FIG. 1 ,one direction of diversion for diverter 30, is to return an object tothe infeed area 12.

Systems of various embodiments provide numerous advantages because ofthe inherent dynamic flexibility. The flexible correspondence betweensorter outputs and destinations provides that there may be fewer sorteroutputs than destinations, so the entire system may require less space.The flexible correspondence between sorter outputs and destinations alsoprovides that the system may choose the most efficient order in which tohandle objects, in a way that varies with the particular mix of objectsand downstream demand. The system is also easily scalable, by addingsorters, and more robust since the failure of a single sorter might behandled dynamically without even stopping the system. It should bepossible for sorters to exercise discretion in the order of objects,favoring objects that need to be handled quickly, or favoring objectsfor which the given sorter may have a specialized gripper.

FIG. 13 shows the destination section 244 (e.g., such as any of sections132 of the system 30) that includes a movable carriage 242 that mayreceive an object 254 from the end effector of the programmable motiondevice. The movable carriage 242 is reciprocally movable between tworows of the destination bins 246 along a guide rail 245. As shown inFIG. 13 , each destination bin 246 includes a guide chute 247 thatguides an object dropped therein into the underlying destination bin246. The carriage 242 moves along a track 245 (as further shown in FIG.14 ), and the carriage may be actuated to drop an object 254 into adesired destination bin 246 via a guide chute 247 (as shown in FIG. 15).

The movable carriage 242 is therefore reciprocally movable between thedestination bins, and the/each carriage moves along a track, and may beactuated to drop an object into a desired destination bin 224. Thedestination bins may be provided in a conveyor (e.g., rollers or belt),and may be biased (for example by gravity) to urge all destination binstoward one end (for example, the distal end). When a destination bin isselected for removal (e.g., because the bin is full or otherwise readyfor further processing), the system will urge the completed bin onto anoutput conveyor to be brought to a further processing or shipmentstation. The conveyor may be biased (e.g., by gravity) or powered tocause any bin on the conveyor to be brought to an output location.

FIGS. 16A and 16B show a bin 251 being urged from the plurality ofdestination bins 246, onto the output conveyor 248 by the use of adisplacement mechanism 255. In accordance with further embodiments, thedestination bins may be provided as boxes or containers or any othertype of device that may receive and hold an item, including box trayassemblies as discussed below.

Following displacement of the bin 251 onto the conveyor 248 (as shown inFIG. 17 ), each of the remaining destination bins may be urged together(as shown in FIG. 18 ) and the system will record the change in positionof any of the bins that moved. This way, a new empty bin may be added tothe end, and the system will record the correct location and identifiedprocessing particulars of each of the destination bins.

As noted above, the bins 246 may be provided as boxes, totes, containersor any other type of device that may receive and hold an item. Infurther embodiments, the bins may be provided in uniform trays (toprovide consistency of spacing and processing) and may further includeopen covers that may maintain the bin in an open position, and mayfurther provide consistency in processing through any of spacing,alignment, or labeling.

For example, FIG. 19 shows an exploded view of a box tray assembly 330.As shown, the box 332 (e.g., a standard shipping sized cardboard box)may include bottom 331 and side edges 333 that are received by a topsurface 335 and inner sides 337 of a box tray 334. The box tray 334 mayinclude a recessed (protected) area in which a label or otheridentifying indicia 346 may be provided, as well as a wide and smoothcontact surface 351 that may be engaged by an urging or removalmechanism as discussed below.

As also shown in FIG. 19 , the box 332 may include top flaps 338 that,when opened as shown, are held open by inner surfaces 340 of the boxcover 336. The box cover 336 may also include a recessed (protected)area in which a label or other identifying indicia 345 may be provided.The box cover 336 also provides a defined rim opening 342, as well ascorner elements 344 that may assist in providing structural integrity ofthe assembly, and may assist in stacking un-used covers on one another.Un-used box trays may also be stacked on each other.

The box 332 is thus maintained securely within the box tray 134, and thebox cover 136 provides that the flaps 338 remain down along the outsideof the box permitting the interior of the box to be accessible throughthe opening 342 in the box cover 336. FIG. 20 shows a width side view ofthe box tray assembly 330 with the box 332 securely seated within thebox tray 334, and the box cover holding open the flaps 338 of the box332. The box tray assemblies may be used as any or both of the storagebins and destination bins in various embodiments of the presentinvention. In various embodiments, the bins or boxes may further includea collection bag in the bin or box prior to receiving objects.

With reference to FIGS. 21A-21D, a box kicker 384 in accordance with anembodiment of the present invention may be suspended by and travel alonga track 386, and may include a rotatable arm 388 and a roller wheel 390at the end of the arm 388. With reference to FIGS. 21B-21D, when theroller wheel 390 contacts the kicker plate 351 (shown in FIG. 19 ) of abox tray assembly 320, the arm 388 continues to rotate, urging the boxtray assembly 380 from a first conveyor 382 to a second conveyor 380.Again, the roller wheel 390 is designed to contact the kicker plate 351of a box tray assembly 381 to push the box tray assembly 381 onto theconveyor 380. Such a system may be used to provide that boxes that areempty or finished being unloaded may be removed (e.g., from conveyor382), or that boxes that are full or finished being loaded may beremoved (e.g., from conveyor 382). The conveyors 380, 382 may also becoplanar, and the system may further include transition roller 383 tofacilitate movement of the box tray assembly 381, e.g., by beingactivated to pull the box tray over to the conveyor 380.

Systems of the invention are highly scalable in terms of sorts-per-houras well as the number of storage bins and destination bins that may beavailable. The system provides in a specific embodiment an input systemthat interfaces to the customer's conveyors and containers, storesobjects for feeding into the system, and feeds those objects into thesystem at a moderate and controllable rate. In one embodiment, theinterface to the customer's process takes the form of a dumper from aGaylord, but many other embodiments are possible. In one embodiment,feeding into the system is by an inclined cleated conveyor with overheadflow restrictors, e.g., baffles. In accordance with certain embodiments,the system feeds objects in at a modest controlled rate. Many optionsare available, including variations in the conveyor slope and speed, thepresence, size and structure of cleats and baffles, and the use ofsensors to monitor and control the feed rate.

The system includes in a specific embodiment a primary perception systemthat monitors the stream of objects on the primary conveyor. Wherepossible the primary perception system may identify the object to speedor simplify subsequent operations. For example, knowledge of the objectson the primary conveyor may enable the system to make better choicesregarding which objects to move to provide a singulated stream ofobjects.

With reference to FIG. 22 , a sortation process of the invention at asorting station may begin (step 400) by providing a singulated stream ofobjects that, one at a time, drop an object into the drop scanner (step402). The system then identifies the new object (step 404). The systemthen will determine whether the object is yet assigned to any collectionbin (step 406). If not, the system will determine whether a next bin isavailable (step 408). If no next bin is available (step 410), therobotic system will return the object to the input buffer (step 410) andreturn to step 402. Alternatively, the system can pick one of thecollection bins that is in process and decide that it can be emptied tobe reused for the object in hand, at which point the control system canempty the collection bin or signal a human worker to do it. If a nextbin is available (and the system may permit any number of bins perstation), the system will then assign the object to a next bin (step412). The system then places the object into the assigned bin (step414). The system then returns to step 402 until finished. Again, incertain embodiments, the secondary conveyor may be an indexed conveyorthat moves in increments each time an object is dropped onto theconveyor. The system may then register the identity of the object,access a warehouse manifest, and determine an assigned bin location orassign a new bin location.

A process of the overall control system is shown, for example, in FIG.23 . The overall control system may begin (step 500) by permitting a newcollection bin at each station to be assigned to a group of objectsbased on overall system parameters (step 502) as discussed in moredetail below. The system then identifies assigned bins correlated withobjects at each station (step 504), and updates the number of objects ateach bin at each station (step 506). The system then determines thatwhen a bin is either full or the system expects that the associatedsorting station is unlikely to see another object associated with thebin, the associated sorting station robotic system will then place thecompleted bin onto an output conveyor, or signal a human worker to comeand empty the bin (step 508), and then return to step 502.

Systems of various embodiments provide numerous advantages because ofthe inherent dynamic flexibility. The flexible correspondence betweensorter outputs and destinations provides that there may be fewer sorteroutputs than destinations, so the entire system may require less space.The flexible correspondence between sorter outputs and destinations alsoprovides that the system may choose the most efficient order in which tohandle objects, in a way that varies with the particular mix of objectsand downstream demand. The system is also easily scalable, by addingsorters, and more robust since the failure of a single sorter might behandled dynamically without even stopping the system. It should bepossible for sorters to exercise discretion in the order of objects,favoring objects that need to be handled quickly, or favoring objectsfor which the given sorter may have a specialized gripper.

The operations of the systems described herein are coordinated by thecentral control system 170 as shown in FIGS. 1 and 3 . The centralcontrol system is comprised of one or more workstations or centralprocessing units (CPUs). The correspondence between barcodes, forexample, and outbound destinations is maintained by the central controlsystem in a database called a manifest. The central control systemmaintains the manifest by communicating with a warehouse managementsystem (WMS). If the perception system successfully recognizes a markingon the object, then the object is then identified and forwarded to anassigned destination station 130. Again, if the object is notidentified, the robotic system may divert the object to a humansortation bin 76 to be reviewed by a human.

Those skilled in the art will appreciate that numerous modification andvariations may be made to the above disclosed embodiments withoutdeparting from the spirit and scope of the present invention.

What is claimed is:
 1. An automated processing system for processingobjects, said automated processing system comprising: an inputconveyance system for moving objects from an input area toward aperception system, said input conveyance system including an inputconveyor that travels along an input conveyance vector that includes aninput conveyance horizontal direction component and an input conveyancevertical direction component; said perception system for receivingobjects from the input conveyance system and for providing perceptiondata regarding an object; a primary transport system for receiving theobject from the perception system and for providing transport of theobject along at least a primary transport vector including a primarytransport horizontal direction component that is generally opposite theinput conveyance horizontal direction component; and a diverter systemfor providing the object in a diverter direction that includes adiverter horizontal direction component that is generally orthogonal tothe primary transport horizontal direction component toward one of aplurality of processing locations, wherein the diverter system providesthe object toward one of the plurality of processing locations via oneof a plurality of secondary transport systems, each secondary transportsystem including a reciprocating carriage configured to deliver theobject to one of a plurality of processing bins or boxes, each of theplurality of bins or boxes being provided on a bin or box conveyorsystem that is biased to urge the bins or boxes to one side of the binor box conveyance system.
 2. The automated processing system as claimedin claim 1, wherein the input conveyor of the input conveyance system isa cleated conveyor.
 3. The automated processing system as claimed inclaim 1, wherein the perception system includes a drop perception unitthrough which the object may be dropped.
 4. The automated processingsystem as claimed in claim 1, wherein the primary transport systemincludes a cleated conveyor.
 5. The automated processing system asclaimed in claim 1, wherein the plurality of processing bins or boxesassociated with each of the secondary transport systems is provided astwo rows of bins or boxes on either side of the reciprocating carriage.6. The automated processing system as claimed in claim 5, wherein eachof the plurality of bins or boxes includes a collection bag.
 7. Anautomated processing system for processing objects, said automatedprocessing system comprising: an input conveyance system for movingobjects from an input area toward a perception system, said inputconveyance system including a conveyor that travels along an inputconveyance vector that includes an input conveyance horizontal directioncomponent, said perception system for receiving objects from the inputconveyance system and for providing perception data regarding an object;a primary transport system for receiving the object from the perceptionsystem and for providing transport of the object along at least aprimary transport vector including a primary transport horizontaldirection component that is generally opposite the input conveyancehorizontal direction component; and a diverter system for providing theobject in a diverter direction that includes a diverter horizontaldirection component that is generally orthogonal to the primarytransport horizontal direction component toward one of a plurality ofprocessing locations; each processing location being provided as atleast two rows of processing locations that each extend along aprocessing location direction that is generally parallel with thehorizontal direction component of the input conveyance system.
 8. Theautomated processing system as claimed in claim 7, wherein the inputconveyor of the input conveyance system includes a cleated conveyor. 9.The automated processing system as claimed in claim 7, wherein theperception system includes a drop perception unit through which theobject may be dropped.
 10. The automated processing system as claimed inclaim 7, wherein the primary transport system includes a cleatedconveyor.
 11. The automated processing system as claimed in claim 7,wherein the diverter system provides the object toward one of theplurality of processing locations via one of a plurality of secondarytransport systems.
 12. The automated processing system as claimed inclaim 11, wherein each secondary transport system includes areciprocating carriage.
 13. The automated processing system as claimedin claim 12, wherein each reciprocating carriage of each secondarytransport system is configured to deliver the object to one of aplurality of processing bins or boxes.
 14. The automated processingsystem as claimed in claim 13, wherein the plurality of destinationstations associated with each of the secondary transport systems isprovided as two rows of bins or boxes on either side of each secondarytransport system.
 15. The automated processing system as claimed inclaim 14, wherein each of the bins or boxes includes a collection bag.16. The automated processing system as claimed in claim 7, wherein theeach of the bins or boxes is provided on a bin or box conveyor systemthat is biased to urge the bins or boxes in the bin or box conveyancesystem to one side of the bin or box conveyance system.
 17. A method ofprocessing objects, said method comprising: moving objects from an inputarea using an input conveyance system toward a perception system in atleast an input conveyance vector that includes an input conveyancehorizontal direction component and an input conveyance verticaldirection component; receiving the objects from the input conveyancesystem and providing perception data regarding an object using a primaryperception system; receiving the object from the primary perceptionsystem and for providing transport of the object using a primarytransport system along at least a first primary transport vector thatincludes a first primary transport horizontal direction component thatis generally parallel with the input conveyance horizontal directioncomponent; diverting the object to one of a plurality of processinglocations in a diverter direction that includes a diverter horizontaldirection component that is generally orthogonal to the input conveyancehorizontal direction component; and moving the object toward one of aplurality of processing locations, each processing location beingprovided as at least two rows of processing locations that each extendalong a processing location direction that is generally parallel withthe horizontal direction component of the input conveyance system. 18.The method as claimed in claim 17, wherein the input conveyance systemincludes a cleated conveyor.
 19. The method as claimed in claim 17,wherein the perception data is provided by a drop perception unitthrough which the object may be dropped.
 20. The method as claimed inclaim 17, wherein the primary transport system includes a cleatedconveyor.
 21. The method as claimed in claim 17, wherein the moving theobject to one of the plurality of processing locations includes usingone of a plurality of secondary transport systems.
 22. The method asclaimed in claim 21, wherein each secondary transport system includes areciprocating carriage.
 23. The method as claimed in claim 22, whereineach reciprocating carriage of each secondary transport system isconfigured to deliver the object to one of the plurality of processingbins or boxes.
 24. The method as claimed in claim 23, wherein theplurality of destination stations associated with each of the secondarytransport systems is provided as two rows of bins or boxes on eitherside of each secondary transport system.
 25. The method as claimed inclaim 24, wherein each of the bins or boxes includes a collection bag.